Information processing apparatus and image generating method

ABSTRACT

A motion detecting section 30 detects the attitude of a head-mounted display device worn on the head of a user. An image generating section 34 generates an image in accordance with the detected attitude of the head-mounted display device. An image providing section 36 provides the head-mounted display device with the generated image. When the attitude of the head-mounted display device is changed during display of the image including an information element, the image generating section 34 displays the information element in such a manner that the information element moves along with the image if the amount of change in the attitude is smaller than a predetermined first angle. The image generating section 34 further displays the information element in a position moved by a predetermined second angle in the image if the amount of change in the attitude reaches the predetermined first angle.

TECHNICAL FIELD

The present invention relates to technology for generating an image tobe displayed on a head-mounted display device.

BACKGROUND ART

A head-mounted display device (HMD) worn on the head of a user providesthe user with a virtual reality (VR) world. Recently, applications havebeen introduced which allow the user to play games while watching thescreen displayed on the HMD. Existing non-portable display devices suchas TV sets let the user's visual field range stray outside the screen.This may prevent the user from concentrating on the screen or having thesense of immersion in the ongoing game. In contrast, wearing the HMDallows the user to watch only images displayed on the HMD. This enhancesthe sense of immersion in the game and also boosts the sense ofentertainment for the user. When the HMD is provided with a headtracking function that updates the display screen in keeping with theattitude of the user's head, the sense of immersion in the visual wordis further enhanced.

Recent years have witnessed widespread use of an omnidirectional camera(omniazimuth camera) for capturing an omnidirectional, 360-degreepanoramic image. Also, development has been underway of remotelycontrollable unmanned flying objects. Mounting multiple cameras on sucha flying object makes it possible to capture an omnidirectionalpanoramic image from the air. If the HMD is caused to display theomnidirectional panoramic image captured in this manner and to let thehead tracking function update the display screen in keeping with theattitude of the user's head, the user is expected to feel as if he orshe is in the real world.

CITATION LIST Patent Literature

[PTL 1]

JP 2015-95045A

SUMMARY Technical Problem

In a recent trend of improving the realistic sensation of imagery,development is actively underway of techniques for widening the viewingangle of the HMD. HMDs with their viewing angle exceeding 80 degreeshave already been commercialized. The wide-viewing-angle HMD offers theuser the world of video imagery comparable to the real world.

The inventors experimented with various kinds of image display on thewide-viewing-angle HMD. The experiments revealed the possibility thatthe user may feel sick depending on how the image is presented. Thissensation will be referred to as “motion sickness” or “simulatorsickness” hereunder in this description. In some experiments, the sameimage did not provoke motion sickness when being viewed on the TV screenbut triggered the unpleasant sensation while being watched on a HMDscreen. Where information elements such as a menu were included in thedisplay image, the user had a feeling of discomfort depending on how theinformation elements were presented. The findings from such diverseexperiments have led the inventors to devise a manner of display controlsuitable for the HMD.

The present invention has been made in view of the above circumstances.An object of the invention is therefore to provide technology forimplementing display control suitable for the HMD.

Solution to Problem

In solving the above problem and according to one aspect of the presentinvention, there is provided an information processing apparatusincluding: a detecting section configured to detect the attitude of ahead-mounted display device worn on the head of a user; an imagegenerating section configured to generate an image in accordance withthe attitude of the head-mounted display device detected by thedetecting section; and an image providing section configured to providethe head-mounted display device with the generated image. The imagegenerating section includes an information element into the generatedimage. When the attitude of the head-mounted display device is changedduring display of the image including the information element on thehead-mounted display device, the image generating section displays theinformation element in such a manner that the information element movesalong with the image if the amount of change in the attitude is smallerthan a predetermined first angle. The image generating section furtherdisplays the information element in a position moved by a predeterminedsecond angle in the image if the amount of change in the attitudereaches the predetermined first angle.

According to another aspect of the present invention, there is providedan image generating method including the steps of: detecting theattitude of a head-mounted display device worn on the head of a user;and generating an image in accordance with the detected attitude of thehead-mounted display device. The image generating step includes aninformation element into the generated image. When the attitude of thehead-mounted display device is changed during display of the imageincluding the information element on the head-mounted display device,the image generating step displays the information element in such amanner that the information element moves along with the image if theamount of change in the attitude is smaller than a predetermined firstangle. The image generating step further displays the informationelement in a position moved by a predetermined second angle in the imageif the amount of change in the attitude reaches the predetermined firstangle.

Incidentally, if other combinations of the above-outlined composingelements or the above expressions of the present invention are convertedbetween different forms such as a method, a device, a system, a computerprogram, and a recording medium having the computer program recordedreadably thereon, they still constitute effective embodiments of thisinvention.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram illustrating a typical configuration of aninformation processing system as one embodiment of the presentinvention.

FIG. 2 is an explanatory diagram explaining a rendering processperformed by an information processing apparatus.

FIG. 3 is a schematic diagram illustrating a typical external shape ofthe HMD.

FIG. 4 is a block diagram illustrating functional blocks of the HMD.

FIG. 5 is a block diagram illustrating functional blocks of theinformation processing apparatus.

FIG. 6 is an explanatory diagram explaining panoramic image data to bestored in an image storing section.

FIG. 7 is a schematic diagram illustrating a display image.

FIG. 8 is a schematic diagram illustrating another display image.

FIG. 9 is a schematic diagram illustrating another display image.

FIG. 10 is a set of schematic diagrams illustrating the top and the rearside of an input device.

FIG. 11 is a schematic diagram illustrating another display image.

FIG. 12 is a set of schematic diagrams illustrating display images givenbefore and after a switch.

FIG. 13 is a set of schematic diagrams illustrating how informationelements are typically superimposed on display images.

FIG. 14 is a set of schematic diagrams illustrating how informationelements are moved along with panoramic images.

FIG. 15 is a set of schematic diagrams illustrating how informationelements are superimposed on panoramic images.

FIG. 16 is a schematic diagram illustrating how information elements aresuperimposed on a panoramic image.

DESCRIPTION OF EMBODIMENT

FIG. 1 illustrates a typical configuration of an information processingsystem 1 as one embodiment of the present invention. The informationprocessing system 1 includes an information processing apparatus 10, ahead-mounted display device (HMD) 100 worn on the head of the user, aninput device 6 operated by the user with fingertips, an imaging device 7for imaging the user wearing the HMD 100, and an output device 4 fordisplaying images.

In this embodiment, the information processing apparatus 10 includes aprocessing device 12 and an output control device 14. The processingdevice 12 is a terminal device that executes diverse applicationsincluding games in response to operation information coming from theinput device 6 operated by the user. The processing device 12 and theinput device 6 may be connected with each other by cable or by knownwireless communication technology. The output control device 14 is aprocessing unit that outputs image data to the HMD 100. The outputcontrol device 14 and the HMD 100 may be connected with each other bycable or by known wireless communication technology.

The imaging device 7 captures an image of the user wearing the HMD 100and feeds the captured image to the processing device 12. The imagingdevice 7 may be a stereo camera. As will be discussed later, the HMD 100is furnished with markers (light-emitting diodes (LEDs) for tracking)permitting tracking of the user's head. In accordance with the capturedmarker positions, the processing device 12 detects the movement of theHMD 100. The HMD 100 is further provided with attitude sensors (anacceleration sensor and a gyro sensor). The processing device 12acquires from the HMD 100 sensor information detected by the attitudesensors, implementing a highly precise tracking process with the jointuse of the sensor information and captured marker images.

Installation of the output device 4 is not mandatory for the informationprocessing system 1 because the user views images displayed on the HMD100. Still, the output control device 14 or the processing device 12 maycause the output device 4 to output the same image as that displayed onthe HMD 100. This allows another user to view on the output device 4 theimage being watched by the user on the HMD 100. As will be describedlater, the image displayed on the HMD 100 is corrected for thedistortion of optical lenses. This requires getting the output device 4to output an image not corrected for the distortion.

In the information processing system 1, the processing device 12, outputdevice 4, input device 6, and imaging device 7 may constitute aconventional game system. In this case, the processing device 12 may bea game device that executes applications such as video games, and theinput device 6 may be a game controller, a keyboard, a mouse, ajoystick, or some other suitable device for supplying the processingdevice 12 with the operation information from the user. Supplementingthis game system with the output control device 14 and the HMD 100 asadditional components makes up the information processing system 1 thatexecutes virtual reality (VR) applications.

The function of the output control device 14 may be incorporated in theprocessing device 12 as part of the functions of VR applications. Thatis, the information processing apparatus 10 may be formed by theprocessing device 12 alone or by the processing device 12 and the outputcontrol device 14. In the description that follows, the function of theprocessing device 12 and that of the output control device 14 necessaryfor implementing VR applications will be explained collectively asfunctions of the information processing apparatus 10.

The information processing apparatus 10 generates image data to bedisplayed on the HMD 100. In this embodiment, the information processingapparatus 10 prepares an omnidirectional, 360-degree panoramic imagecaptured by an omnidirectional camera, and causes the HMD 100 to displayan image generated on the basis of a visual line direction determined bythe attitude of the HMD 100 worn on the user's head. The content ofdisplay may be a still image or a moving image. The display content isnot limited to actually captured images; the content may also be imagesrendered in real time by a game application.

The HMD 100 is a display device that is worn on the user's head anddisplays images through optical lenses on a display panel positioned infront of the user's eyes. The HMD 100 displays a left-eye image on theleft half of the display panel and a right-eye image on the right halfof the display panel in a manner independent of each other. These imagesconstitute parallax images viewed from the right and left viewpoints.When displayed on the right-half and left-half portions of the displaypanel, the parallax images make up a stereoscopic image. Because theuser views the display panel through optical lenses, the informationprocessing apparatus 10 supplies the HMD 100 with the image datacorrected beforehand for the optical distortion of the lenses. In theinformation processing apparatus 10, the optical distortion correctingprocess may be performed by the processing device 12 or by the outputcontrol device 14.

FIG. 2 is an explanatory diagram explaining a rendering processperformed by the information processing apparatus 10. For VR applicationof this embodiment, a virtual environment is provided in such a mannerthat the user feels as if he or she is at the center of a sphere andthat the image visible to the user is varied when his or her visual linedirection is changed. A content image constituting the image material ispasted on the inner circumferential surface of a virtual spherecentering on a center point 9 at which the user is positioned. Thecontent image is an omnidirectional, 360-degree panoramic image capturedby an omnidirectional camera. This content image is pasted on the innercircumferential surface of the virtual sphere in such a manner that thezenith and nadir positions of the image coincide with those of thevirtual sphere. When the zenith and nadir positions of the user's realworld are thus aligned with those of the visual world offered to the HMD100, it permits implementation of a VR application that reproduces therealistic visual world.

The information processing apparatus 10 detects the rotation angle andinclination of the user's head (HMD 100 in reality) by performing atracking process on the user's head position. The rotation angle of theHMD 100 is defined relative to a horizontal reference direction. Forexample, the direction in which the HMD 100 is oriented when switched onmay be established as the reference direction. The inclination of theHMD 100 is defined as an inclination angle relative to a horizontalplane. Existing techniques may be used to implement the head trackingprocess. The information processing apparatus 10 may detect the rotationangle and inclination of the HMD 100 solely from the sensor informationdetected by the attitude sensors of the HMD 100. The informationprocessing apparatus 10 may also detect the rotation angle andinclination of the HMD 100 with higher accuracy by analyzing images ofthe markers (tracking LEDs) on the HMD 100 captured by the imagingdevice 7.

The information processing apparatus 10 determines the attitude of avirtual camera 8 in the virtual sphere in accordance with the detectedrotation angle and inclination of the HMD 100. The virtual camera 8 isarranged to image the inner circumferential surface of the virtualsphere from its center point 9. The information processing apparatus 10aligns the detected rotation angle and inclination with the rotationangle and inclination of the optical axis of the virtual camera 8 in thevirtual sphere. The information processing apparatus 10 acquires acaptured image 5 from the virtual camera 8, i.e., performs a renderingprocess, corrects the image for the optical distortion of the opticallenses, and feeds the image data to the HMD 100. Whereas FIG. 2illustrates a single virtual camera 8, a right-eye virtual camera 8 anda left-eye virtual camera 8 are in reality provided to generate theirrespective image data.

FIG. 3 illustrates a typical external shape of the HMD 100. In thisexample, the HMD 100 is made up of an output mechanism section 102 and awearing mechanism section 104. The wearing mechanism section 104includes a wearing band 106 worn by the user to fasten the HMD 100around the head. The wearing band 106 uses a material or a structurethat allows the band to be adjusted in length to fit around the user'shead.

The output mechanism section 102 includes an enclosure 108 that coversboth eyes of the user wearing the HMD 100. Inside the enclosure 108 is adisplay panel positioned straight against the eyes when the HMD 100 isworn. The display panel may be formed by a liquid crystal panel or by anorganic electroluminescent (EL) panel, for example. Also inside theenclose 108 are a pair of right and left optical lenses positionedbetween the display panel and the user's eyes when the HMD 100 is worn,the optical lenses serving to widen the viewing angle of the user. TheHMD 100 may be further equipped with speakers or earphones positionedagainst the user's ears when the HMD 100 is worn.

The external surface of the enclosure 108 is equipped withlight-emitting markers 110 a, 110 b, 110 c and 110 d. Although trackingLEDs make up the light-emitting markers 110 in this example, othersuitable markers may be used instead. In any case, the markers need onlybe imaged by the imaging device 7 in a manner permitting image analysisby the information processing apparatus 10. Whereas the light-emittingmarkers 110 are not limited in number and in arrangement, their numberand their arrangement need to be such as to permit detection of theattitude of the HMD 100 (its rotation angle and inclination). In theillustration, the markers are arranged at four corners on the front ofthe enclosure 108. The light-emitting markers 110 may be additionallyarranged on the sides and the rear of the wearing band 106. Theseadditional markers may be imaged if the user turns his or her back tothe imaging device 7.

The HMD 100 may be connected with the information processing apparatus10 by cable or by known wireless communication technology. The HMD 100transmits sensor information detected by the attitude sensors to theinformation processing apparatus 10. The HMD 100 also receives imagedata generated by the information processing apparatus 10 and displaysthe received data on the display panel.

Whereas the HMD 100 illustrated in FIG. 3 is an immersive(non-transmissive) display device that fully covers both eyes, the HMD100 may also be a transmissive display device. In terms of shape, theHMD 100 may be of hat type as illustrated but may also be of spectacletype.

FIG. 4 illustrates functional blocks of the HMD 100. A control section120 is a main processor which processes diverse data such as image data,audio data, sensor information, and instructions and which outputs whatis processed. A storage section 122 temporarily stores the data andinstructions processed by the control section 120. Attitude sensors 124detect attitude information about the HMD 100 such as its rotation angleand inclination. The attitude sensors 124 include at least a three-axisacceleration sensor and a three-axis gyro sensor. A microphone 126converts the user's voice to an electrical signal. The light-emittingmarkers 110 are multiple LEDs attached to the wearing band 106 andenclosure 108 of the HMD 100.

A communication control section 128 transmits the data input from thecontrol section 120 to an external information processing apparatus 10by wired or wireless communication via a network adapter or an antenna.The communication control section 128 further receives data from theinformation processing apparatus 10 by wired or wireless communicationvia the network adapter or the antenna, and outputs the received data tothe control section 120.

Upon receipt of image data and audio data from the informationprocessing apparatus 10, the control section 120 feeds the received datato the display panel 130 for display and to an audio output section 132for audio output. The control section 120 further causes thecommunication control section 128 to transmit the sensor informationfrom the attitude sensors 124 and the audio data form the microphone 126to the information processing apparatus 10.

FIG. 5 illustrates functional blocks of the information processingapparatus 10. The information processing apparatus 10 includes a sensorinformation acquiring section 20, a captured image acquiring section 22,and an instruction acquiring section 24, the sections serving as aninput interface to the outside. The sensor information acquiring section20 acquires the sensor information at predetermined intervals from theattitude sensors 124 of the HMD 100. The captured image acquiringsection 22 acquires captured images from the imaging device 7 imagingthe HMD 100 at predetermined intervals. For example, the imaging device7 captures images at intervals of 1/60 second, while the captured imageacquiring section 22 acquires captured images at intervals of 1/60second. The instruction acquiring section 24 acquires instructions inputby the user from the input device 6.

The information processing apparatus 10 further includes a motiondetecting section 30, a visual line direction determining section 32, animage generating section 34, and an image providing section 36. Themotion detecting section 30 detects the attitude of the HMD 100 worn onthe user's head. The visual line direction determining section 32determines the visual line direction in accordance with the attitude ofthe HMD 100 detected by the motion detecting section 30. The imagegenerating section 34 generates images in accordance with the detectedattitude of the HMD 100. Specifically, the image generating section 34generates images based on the visual line direction determined by thevisual line direction determining section 32. The image providingsection 36 provides the generated image to the HMD 100.

The components indicated in FIG. 5 as functional blocks for carrying outdiverse processes may each be configured by a circuit block, a memory,or some other large-scale integrated (LSI) circuit in terms of hardware,or by a program loaded into a memory in terms of software. Thus it willbe understood by those skilled in the art that these functional blocksare realized by hardware alone, by software alone, or by a combinationof both in diverse forms and are not limited to any of such forms.

An image storing section 40 stores 360-degree panoramic image datacaptured beforehand. The image storing section 40 may store multiplecontent images. The content images may be still images or moving images.The image storing section 40 in the embodiment stores omnidirectionalpanoramic image data. The information processing apparatus 10 providesthe user wearing the HMD 100 with an omnidirectional panoramic image.When the user turns his or her head to the right or to the left (theuser may turn his or her entire body right or left) to turn his or herhorizontal visual line right or left, the display panel 130 of the HMD100 displays a panoramic image in the right or left direction. When theuser tilts his or her head upward or downward to incline the visual linevertically, the display panel 130 of the HMD 100 displays a panoramicimage in the upward or downward direction.

An information element storing section 42 stores information elements tobe included in panoramic images. For example, the information elementsmay be menu items for changing the luminance of content images or forselecting a content image as an image material. When the user inputs tothe input device 6 an instruction to display information elements, theimage generating section 34 includes the information elements into theimage to be generated. The information elements superimposed on thepanoramic image are then displayed on the display panel 130 of the HMD100. The information elements may also be notification informationdestined for the user. In this case, the information elements may besuperimposed on the panoramic image without being invoked by the useroperating the input device 6.

FIG. 6 is an explanatory diagram explaining panoramic image data to bestored in the image storing section 40. For purpose of explanation, FIG.6 illustrates part of an omnidirectional panoramic image, and omitsportions of the image from the horizontal plane downward as well asportions of the image in the right and left directions. As discussedabove with reference to FIG. 2, the display panel 130 of the HMD 100displays an image formed by rendering the panoramic image pasted on theinner circumferential surface of the virtual sphere. When the userchanges his or her visual line direction by moving the rotation angleand inclination of the HMD 100, the panoramic image being displayed ismoved in keeping with the visual line direction.

The communication control section 128 of the HMD 100 transmits to theinformation processing apparatus 10 the sensor information acquired atpredetermined intervals by the attitude sensors 124. The imaging device7 images the HMD 100 at predetermined intervals and transmits thecaptured images to the information processing apparatus 10. Referring toFIG. 5, the sensor information acquiring section 20 acquires the sensorinformation from the attitude sensors 124 and feeds the acquiredinformation to the motion detecting section 30. The captured imageacquiring section 22 acquires a captured image and feeds it to themotion detecting section 30.

In detecting the attitude of the HMD 100, the motion detecting section30 performs the head tracking process to detect the attitude of theuser's head wearing the HMD 100. The head tracking process is carriedout to synchronize the field of view displayed on the display panel 130of the HMD 100 with the attitude of the user's head. The head trackingprocess of the embodiment involves detecting the rotation angle of theHMD 100 relative to a horizontal reference direction and the inclinationangle of the HMD 100 relative to a horizontal plane. The horizontalreference direction may be established as the direction in which the HMD100 is oriented when switched on, for example.

The head tracking process may be performed using existing techniques,with the motion detecting section 30 detecting the rotation angle of theHMD 100 relative to the horizontal reference direction and theinclination angle of the HMD 100 relative to the horizontal plane solelyfrom the sensor information given by the attitude sensors 124.Preferably, the accuracy of the detection may be enhanced by theadditional use of captured images of the light-emitting markers 110 fortracking purposes. The motion detecting section 30 detects the rotationangle and the inclination angle at predetermined intervals. For example,if the image fed to the HMD 100 is captured at 60 frames per second(fps), the motion detecting section 30 may preferably perform itsdetecting process at intervals of 1/60 second.

The visual line direction determining section 32 determines the visualline direction in accordance with the attitude of the HMD 100 detectedby the motion detecting section 30. The visual line direction thusdetermined is both the visual line direction of the user and that(optical axis direction) of the virtual camera 8 arranged at the centerpoint 9 of the virtual sphere (see FIG. 2). Here, the visual linedirection determining section 32 may determine the rotation angle andinclination angle detected by the motion detecting section 30 directlyas representative of the visual line direction (optical axis direction)of the virtual camera 8. The visual line direction determining section32 may also determine the visual line direction of the virtual camera 8by carrying out some suitable correcting process. If the motiondetecting section 30 is not supplied with stable sensor information, aswhen the sensor information is contaminated with noise, the motiondetecting section 30 might detect vibrating movements despite the user'shead being stationary. In such a case, the visual line directiondetermining section 32 may determine the visual line direction bysmoothing out the movements detected by the motion detecting section 30.

Incidentally, the field of view of the humans is verticallyasymmetrical; the view under the visual line is slightly wider than theview thereabove. Thus the visual line direction determining section 32may determine the visual line direction of the virtual camera 8 byslightly tilting down the inclination angle detected by the motiondetecting section 30.

The image generating section 34 generates an image in accordance withthe attitude of the HMD 100 detected by the motion detecting section 30.Specifically, the image generating section 34 generates an image basedon the visual line direction of the virtual camera 8 determined by thevisual line direction determining section 32. The image generatingsection 34 determines the right-eye and left-eye visual fields definedby the visual line direction, before generating a right-eye image and aleft-eye image through rendering. At this point, the image generatingsection 34 generates a panoramic image corrected for the distortion ofthe image light coming from the display panel and passing through theoptical lenses.

The HMD 100 of the embodiment provides the user with the visual field atan angle of approximately 100 degrees to the horizontal direction and atan angle of approximately 100 degrees to the vertical direction.Referring to FIG. 2, the captured image 5 is obtained at an angle ofview of approximately 100 degrees to the horizontal direction and at anangle of view of approximately 100 degrees to the vertical directioninside the virtual sphere. The captured image 5 is displayed on thedisplay panel 130 of the HMD 100. As mentioned above, the visual fieldof the humans is slightly wider under the visual line than thereabove.For this reason, the optical lenses and the display panel 130 of the HMD100 may be tilted 5 degrees in a direction opposite to the eye position.The optical lenses and the display panel 130 may thus be arranged toimplement a vertical field of view at an upward angle of 45 degrees andat a downward angle of 55 degrees.

FIG. 7 illustrates a display image 200 a generated by the imagegenerating section 34. In the drawings referenced below, the displayimage will be presented as an image cut out of the panoramic image forthe ease of understanding the positional relation of the display imagewithin the entire panoramic image.

The image generating section 34 generates an image based on a visualline direction 202 a determined by the visual line direction determiningsection 32. In practice, the image generating section 34 generates aright-eye display image and a left-eye display separately throughrendering as different images each containing a parallax. In thedescription that follows, however, no mention will be made of the imagesbeing generated separately for both eyes. The image providing section 36provides the HMD 100 with the display image 200 a generated by the imagegenerating section 34. The control section 120 in the HMD 100 causes thedisplay panel 130 to display the display image 200 a. The user is thenable to view the display image 200 a displayed on the display panel 130.

FIG. 8 illustrates a display image 200 b generated by the imagegenerating section 34. The visual line direction determining section 32determines the visual line direction in accordance with the attitude ofthe HMD 100. The image generating section 34 generates the image basedon the visual line direction thus determined. This example indicatesthat the user has turned his or her head to the left, with the visualline changed continuously from a visual line direction 202 a to a visualline direction 202 b. Here, the user has turned his or her headapproximately 60 degrees to the left. The turning motion causes theimage generating section 34 to generate images at intervals of 1/60second in a manner turning the panoramic image continuously to the leftstarting from the display image 200 a. The image providing section 36provides the HMD 100 with the generated images at intervals of 1/60second.

FIG. 9 illustrates a display image 200 c generated by the imagegenerating section 34. The visual line direction determining section 32determines the visual line direction in accordance with the attitude ofthe HMD 100. The image generating section 34 generates the image basedon the visual line direction thus determined. This example indicatesthat the user has tilted up his or her head from the state in which thedisplay image 200 a is displayed on the display panel 130, with thevisual line changed continuously from the visual line direction 202 a toa visual line direction 202 c. Here, the user has tilted up his or herhead approximately 30 degrees. The tilting motion causes the imagegenerating section 34 to generate images at intervals of 1/60 second ina manner moving the panoramic image continuously upward starting fromthe display image 200 a. The image providing section 36 provides the HMD100 with the generated images at intervals of 1/60 second.

As described above, the user varies the visual line direction by movinghis or her head. The information processing apparatus 10 provides theHMD 100 with a panoramic image given in a desired direction so that theHMD 100 will display the image on the display panel 130. The manner inwhich the visual line direction is varied by moving the head is the sameas in the real world. What is virtually brought about thus accords withthe user's sensation of the real world. Where the HMD 100 provides theuser with a wide viewing angle, the user's sense of immersion in thepanoramic image is further enhanced.

Meanwhile, if the user wants to view a panoramic image immediatelybehind his or her back, the user obviously has to turn back fully. Justturning the head is not enough for turning back fully; the user needs tochange the orientation of the body. If the user is sitting on anon-rotating chair such as a sofa, the user is required to stand up tochange the orientation of the body. If the HMD 100 is a non-transmissivedisplay device, the user has both eyes covered with the enclosure 108and is unable to see the surroundings. This might make the user slightlyhesitate to move his or her legs to change the orientation of the body.

With this embodiment, the user is allowed to input through the inputdevice 6 instructions to switch the visual line direction; the user canchange the visual line direction by operating the input device 6 withoutmoving his or her head. Whereas the embodiment presents the input device6 as a device different from the HMD 100, the input device 6 mayalternatively be integrated with the HMD 100. That is, the input device6 may be provided as an operation input section attached to the wearingband 106 or the enclosure 108 of the HMD 100, for example.

Subfigure (a) in FIG. 10 illustrates the top of the input device 6. Theuser operates the input device 6 by holding a left-side grip 78 b withthe left hand and a right-side grip 78 a with the right hand. Theenclosure top of the input device 6 is equipped with arrow buttons 71,analog sticks 77 a and 77 b, and operation buttons 76 making up theinput section. The arrow buttons 71 include an upward button 71 a, aleftward button 71 b, a downward button 71 c, and a rightward button 71d. The right analog stick 77 a and the left analog stick 77 b are tiltedto input directions and tilting amounts. The right analog stick 77 a andthe left analog stick 77 b also function as depressable buttons that aredepressed when pushed by the user and return to the initial positionwhen released. On the enclosure top, a touch pad 79 is arranged in aflat region between the arrow buttons 71 and the operation buttons 76.The touch pad 79 also functions as a depressable button that isdepressed when pushed by the user and returns to the initial positionwhen released.

A home button 80 is arranged between the right analog stick 77 a and theleft analog stick 77 b. The home button 80 is used to turn on the powersupply of the input device 6 and to activate a communication functionfor wirelessly communicating with the information processing apparatus10 at the same time. A SHARE button 81 is arranged to the left of thetouch pad 79. The SHARE button 81 is used to input the user'sinstructions to the operating system (OS) of the information processingapparatus 10 or to its software. An OPTIONS button 82 is arranged to theright of the touch pad 79. The OPTIONS button 82 is used to input theuser's instructions to an application (game) executed by the informationprocessing apparatus 10. The SHARE button 81 and the OPTIONS button 82may each be formed as a push-button.

Subfigure (b) in FIG. 10 illustrates the rear side of the input device6. At the upper part of the rear side of the enclosure of the inputdevice 6 is a bent touch pad 79 extending from the top of the enclosure.At the lower part of the rear side of the enclosure is a horizontallyelongated light-emitting section 85. The light-emitting section 85 hasred (R), green (G), and blue (B) LEDs that are lit in accordance withluminescent color information sent from the information processingapparatus 10. An upper-right R1 button 83 a and an upper-left L1 button83 b are each configured as a push-button. A lower-right R2 button 84 aand a lower-left L2 button 84 b are each configured as apivotally-supported trigger type button.

Of the operating members of the input device 6 in this embodiment, theleft analog stick 77 b is used to input an instruction to switch thevisual line direction. Tilting the left analog stick 77 b left moves thevisual line direction to the left, and tilting it right moves the visualline direction to the right. The left analog stick 77 b is thus fit forthe user to intuitively change the visual line direction. Otheroperating members such as the right analog stick 77 a or the arrowbuttons 71 may be used alternatively to input the instruction to switchthe visual line direction.

The inventors experimented with various techniques of the process forchanging the visual line direction by operation of the left analog stick77 b. One technique of the changing process involved continuously movingthe visual line direction in accordance with the direction in which theleft analog stick 77 b is tilted to generate images in such a mannerthat the panoramic image would be moved in a flowing manner. This is acommonly-practiced method of changing the visual line direction in gamesdisplayed on the TV screen and is thought to well accord with the user'ssenses. Diverse experiments were also made with the speed at which theimage was moved.

The experiments revealed that moving the panoramic image continuously byoperation of the left analog stick 77 b gives motion sickness to theuser watching the image. Motion sickness was provoked regardless of thespeed at which the image was moved. People experienced motion sickness,to a greater or lesser extent, regardless of whether the image was movedquickly or slowly.

On the TV screen, the display image is often moved continuously in onedirection for dramatic impact. At such a time, the user usually does notexperience motion sickness. Looking into this point, the inventors foundthat a difference in viewing angle is one of the causes of theunpleasant sensation.

The user views the TV screen from a certain distance. That means theviewing angle of the user in the horizontal direction of the TV screenis several dozen degrees at most. In contrast, the user's viewing angleon a wide-viewing-angle HMD is close to 90 degrees or equal to orgreater than 90 degrees (the viewing angle in the horizontal directionof the HMD 100 is assumed to be approximately 100 degrees hereunder).That is, the HMD 100 tends to give motion sickness to the user becausethe image is continuously moved within the range of approximately 100degrees (or over the entire field of view). This turned out to be aproblem unique to the HMD 100. The inventors then experimented withmoving the panoramic image not continuously but discontinuously byoperation of the left analog stick 77 b. The discontinuous movement ofthe image was found to prevent the onset of motion sickness. Given thesefindings, upon acquiring the operation of the left analog stick 77 b asan instruction to switch the visual line direction, the informationprocessing apparatus 10 of the embodiment performs display control insuch a manner as to move the panoramic image at intermittent times(i.e., in steps), the image being moved at a predetermined angle perstep.

When the user tilts the left analog stick 77 b on the input device 6,the instruction acquiring section 24 acquires from the input device 6the instruction to switch the visual line direction in the direction inwhich the stick is tilted. For example, if the left analog stick 77 b istilted once and allowed to return immediately to the initial position,the instruction acquiring section 24 acquires an instruction to switchthe visual line direction in one step in the direction of the tilt. Ifthe left analog stick 77 b is tilted continuously, the instructionacquiring section 24 continuously acquires instructions to switch thevisual line direction in the direction of the tilt.

Whether the instruction acquiring section 24 acquires the tiltingoperation of the left analog stick 77 b as a one-time switchinginstruction or as a continuous switching instruction is determineddepending on whether the tilting time (i.e., a time period during whichthe stick is tilted and is allowed to return to the initial position)exceeds a predetermined time period. Specifically, if the tilting timeis less than the predetermined time period, the instruction acquiringsection 24 acquires the tilting operation as a one-time switchinginstruction; if the tilting time reaches the predetermined time period,the instruction acquiring section 24 acquires at that time the tiltingoperation as another switching instruction. If the tilting operation isfurther continued, the switching instruction may be acquired every timea time period shorter than the predetermined time period elapses.

When the instruction acquiring section 24 acquires the instruction toswitch the visual line direction, the visual line direction determiningsection 32 changes the visual line direction by a predetermined angle.For example, if the left analog stick 77 b is tilted to the left, theinstruction acquiring section 24 acquires a switching instruction toturn the visual line direction to the left. The visual line directiondetermining section 32 then changes the visual line direction to theleft by the predetermined angle. Explained below is an example in which,with the display panel 130 of the HMD 100 displaying the display image200 a illustrated in FIG. 7, the user inputs to the input device 6 aswitching instruction to turn the visual line direction to the left.

FIG. 11 illustrates a display image 200 d generated by the imagegenerating section 34. When the instruction acquiring section 24acquires the switching instruction to turn the visual line direction tothe left, the visual line direction determining section 32 turns thevisual line direction to the left by the predetermined angle. Thepredetermined angle is assumed to be larger than 10 degrees and smallerthan the viewing angle (100 degrees) of the HMD 100 in the horizontaldirection.

If the angle to be changed in one step is set to be smaller than 10degrees, at least 18 steps are needed by the user to view the imageimmediately behind his or her back (i.e., to make a 180-degree turn),for example. The numerous steps required might make the user a littleimpatient. On the other hand, if the angle to be changed in one step islarger than the viewing angle of the HMD 100, then a totally new image(an image not overlapping with the unswitched image) would suddenlyappear on the display panel 130; the continuity with the unswitchedimage is not guaranteed. Preferably, the angle to be changed in one stepis set to be larger than 10 degrees and smaller than the viewing angleof the HMD 100.

More preferably, the angle to be changed in one step may be set to beless than half the viewing angle of the HMD 100. This causes at leasthalf of the unswitched image to be included in the newly switched image.The user is then able to recognize image continuity before and after theswitch. To let the user better recognize image continuity, the angle tobe changed in one step may preferably be smaller. After experimentingwith diverse angles to be changed per step, the inventors concluded thatan angle larger than 15 degrees and smaller than 30 degrees is mostpreferred as the angle to be changed per step in view of imagecontinuity and the time to reach the desired angle through continuousswitches. FIG. 11 illustrates an example in which the viewing angle ischanged by 22.5 degrees in one step.

Generally, the central visual field of the humans is said to be 40 to 45degrees and the peripheral visual field to be approximately 200 degrees.With the characteristic of the central visual field taken intoconsideration and with the changing angle per step set to be smallerthan the central visual field, the same image may be arranged topartially remain in the central visual field before and after theswitch. In view of this, it makes sense to set the angle to be changedin one step at 30 degrees or less. Preferably, the angle to be changedin one step is set to be a value obtained by dividing 360 degrees by aninteger N so that the user can return to the initial display position byinputting through the input device 6 the integer number of switchinginstructions.

FIG. 12 illustrates display images displayed on the display panel 130before and after a switch. Subfigure (a) in FIG. 12 illustrates thedisplay image 200 a, and Subfigure (b) in FIG. 12 indicates the displayimage 200 d. The display image 200 d is obtained by turning the displayimage 200 a to the left by 22.5 degrees in one step. Given the viewingangle of 100 degrees, the angle to be changed in one step is set to be22.5 degrees. Thus making one switch changes 22.5 percent of the image,which leaves 77.5 percent of the image being the same before and afterthe switch. This allows the user to easily recognize image continuity.

If the instruction acquiring section 24 acquires switching instructionscontinuously, the visual line direction determining section 32 changesthe visual line direction at intervals of a predetermined angle. Thechanging interval may be a fixed value such as 1 second or defineddynamically depending on the tilting amount of the left analog stick 77b. That is, the larger the tilting amount, the shorter the changinginterval may be set to be.

With this embodiment, the switching instructions given by the use of theinput device 6 are effective only in the horizontal direction and not inthe vertical direction. That is because people, when sitting orstanding, usually look in the horizontal direction and not in the upwardor downward direction continuously and because the actual senses ofpeople are respected in VR applications. It is possible to let theswitching instructions be effective in the vertical direction as well.Whether or not to enable switching instructions in the verticaldirection may be determined depending on the content image. Theswitching instructions in the vertical direction may be enabled forcontent images such as captured images of a starry sky in a celestialsphere where the zenith and nadir positions need not be taken intoaccount.

Whereas FIG. 12 illustrates the display images 200 a and 200 d displayedbefore and after a switch made with the angle of 22.5 degrees in onestep, the user may wish to make the changing angle larger or smallerthan 22.5 degrees. In such a case, the display panel 130 may be causedto display alternatives for selecting the angle to be changed per step.The user may then operate the input device 6 to select one of thealternatives displayed on the display panel 130. This embodimentprovides the user with the alternatives of 15 degrees and 30 degrees inaddition to the choice of 22.5 degrees.

Explained below is a procedure for causing the display panel 130 of theHMD 100 to display the alternatives for selecting the angle to bechanged per step, with the display panel 130 displaying the displayimage 200 a given in Subfigure (a) of FIG. 12.

The user may cause the display panel 130 to display menu items byoperating a predetermined input section of the input device 6. Forexample, a menu display operation may be assigned to a triangle (Δ)button 75 (see Subfigure (a) in FIG. 10). When the user pushes the Δbutton 75, the input device 6 transmits to the information processingapparatus 10 operation information indicating that the Δ button 75 ispushed. The instruction acquiring section 24 in the informationprocessing apparatus 10 acquires the pushing information from the Δbutton 75 as an instruction to display the menu items.

When the instruction acquiring section 24 acquires the instruction todisplay the menu items, the image generating section 34 performs aprocess of including the menu items into the panoramic image beinggenerated. The menu items are an example the information elements to bepresented to the user. Alternatively, the information elements may beother items or notification information. When the image generatingsection 34 includes the information elements such as the menu into thepanoramic image, the user may view the displayed information elements onthe display panel 130 of the HMD 100 while looking at the panoramicimage in the visual line direction on the display panel 130.

The information element storing section 42 stores the informationelements to be included in the panoramic image. The image generatingsection 34 reads from the information element storing section 42 theinformation elements corresponding to the display instruction, andsuperimposes a window listing the information elements onto thepanoramic image. The image providing section 36 provides the HMD 100with the panoramic image including the menu items.

Subfigure (a) in FIG. 13 illustrates an example in which a menu window204 is superimposed on the display image 200 a. In Subfigure (a) of FIG.13, the panoramic image behind the menu window 204 is hidden thereby.Preferably, the menu window 204 in practice may be displayedtransparently so that the panoramic image behind the menu window 204 canbe seen through.

In the menu window 204, a selection rectangle enclosing one item isdisplayed. The user may move the selection rectangle by pushing theupward button 71 a or the downward button 71 c on the input device 6.When the selection rectangle is positioned to the desired item and anenter button (e.g., a circle (∘) button 72) is pushed, the display panel130 displays the information elements subordinate to the selected item.

In the information processing apparatus 10, the instruction acquiringsection 24 acquires the operation information from the arrow buttons 71as instructions to select the item and the operation information fromthe ∘ button 72 as the instruction to enter the item. The imagegenerating section 34 moves the selection rectangle in accordance withthe instructions to select the item, reads from the information elementstoring section 42 the corresponding information elements in accordancewith the instruction to enter the item, and includes the retrievedinformation elements into the panoramic image.

Subfigure (b) in FIG. 13 illustrates an example in which a selectionwindow 206 is superimposed on the display image 200 a. If the item“Change the angle to be changed in one step” is selected in the menuwindow 204 of Subfigure (a) in FIG. 13, the selection window 206 isdisplayed to present alternatives for selecting the angle to be changedper step. Preferably, as with the menu window 204, the selection window206 may be displayed transparently to let the user view the panoramicimage behind the window. When the user selects one of theangle-selecting alternatives and enters the selection on the selectionwindow 206, the new angle to be changed in one step is selected.

The information elements are superimposed on the panoramic image in themanner described above and displayed on the display panel 130. In theinformation processing system 1, the panoramic image on the displaypanel 130 is changed in accordance with the changing attitude of the HMD100 (i.e., movement of the visual line direction) and with operations onthe left analog stick 77 b. The inventors examined how the HMD 100 wouldcontrol the manner in which the information elements are displayed alongwith the panoramic image.

The inventors first experimented with a technique for always displayingthe information elements at the center of the screen on the displaypanel 130. In this case, the position where the information elements aredisplayed on the screen remains unchanged regardless of the panoramicimage behind the information elements being changed in keeping with thechanging visual line direction. The information elements are alwaysdisplayed in a fixed position at the center of the screen. Thisdisplaying technique turned out to give the impression as if theinformation elements were moving within the panoramic image, therebyprovoking a feeling of discomfort.

The inventors then experimented with a technique by which theinformation elements, once included into the panoramic image, are keptfixed where they are arranged. That is, the position where theinformation elements are arranged in the panoramic image is fixed. Whenthe user changes his or her visual line direction, the informationelements are moved integrally with the panoramic image. This techniqueturned out not to provoke a feeling of discomfort in the user becausethe relative positional relation between the information elements andthe panoramic image is fixed so that the information elements appear tobe part of the panoramic image.

In this case, however, a large movement of the visual line direction cancause the information elements to disappear from the display screenbecause they move along with the panoramic image. The user then losessight of the information elements. In particular, because the inputdevice 6 is assigned to the menu operations while the menu is beingdisplayed, the user is required to change his or her visual linedirection to find the menu window that has disappeared, regardless ofwhether the menu operations are to be continued or terminated.

To solve that problem, the inventors came up with a technique forbasically fixing the relative positional relation between informationelements and the panoramic image so that the information elements willmove together with the panoramic image. When the information elementsare about to disappear from the screen, this technique causes therelative positional relation between the information elements and thepanoramic image to be changed.

According to this technique, the point at which information elements areinitially displayed is determined as a reference point. The attitude ofthe HMD 100 is then changed from the reference point. If the amount ofchange in the attitude is smaller than a predetermined first angle, theimage generating section 34 displays the information elements in amanner that they move along with the image. If the amount of change inthe attitude reaches the predetermined first angle, the image isgenerated in such a manner that the information elements are displayedin a position moved by a predetermined second angle within the image.The amount of change in the attitude of the HMD 100 corresponds to theamount of change in the movement of the visual line direction.

Specifically, the visual line direction determining section 32 monitorschanges in the movement of the visual line direction starting from thepoint at which the information elements are initially displayed. If themovement of the visual line direction is smaller than the predeterminedfirst angle, the image generating section 34 displays the informationelements in a manner that they move along with the panoramic image. Ifthe movement of the visual line direction reaches the predeterminedfirst angle, the image generating section 34 displays the informationelements in a position moved by the predetermined second angle withinthe panoramic image. Explained below is an example in which the visualline direction determining section 32 monitors the rotation angle of thevisual line in the horizontal direction, with the image generatingsection 34 performing display control of the information elementsaccordingly.

For example, suppose that starting from the point at which the displayof information elements is started, the user turns his or her head tothe left in the horizontal direction so that the user's visual linedirection is changed to the left. In this case, the panoramic image isdisplayed as if it were flowing to the right. Thus the displayedinformation elements also appear to be flowing rightward on the displaypanel 130. When the traveling angle of the information elements, i.e.,the rotation angle of the visual line in the horizontal direction,reaches the predetermined first angle, the image generating section 34displays the information elements after moving them to a positionreached by a left turn at the predetermined second angle within thepanoramic image. The first angle is set to be such that the informationelements will not disappear completely from the user's visual field whenmoved. The information elements are thus kept displayed at leastpartially on the display panel 130. In the description that follows, thefirst angle may be referred to as “position change reference angle” andthe second angle as “return angle” where appropriate.

Explained below is an example in which the user turns his or her visualline direction to the left from an initial display state where the menuwindow 204 is initially displayed as illustrated in Subfigure (a) ofFIG. 13. FIG. 14 illustrates how the menu window 204 listing theinformation elements is moved along with the panoramic image. When theuser turns his or her head to the left in the horizontal direction, themotion detecting section 30 detects the attitude of the HMD 100. Thevisual line direction determining section 32 turns the visual linedirection to the left in accordance with the detected attitude of theHMD 100. Subfigure (a) in FIG. 14 illustrates the display image 200 d asit is turned to the left by 22.5 degrees from the initial display state.As discussed above, the 22.5-degree turn is made alternatively byoperation of the left analog stick 77 b. Without moving the head, theuser may tilt the left analog stick 77 to the left to have the displayimage 200 d displayed on the display panel 130.

Because the relative positional relation between the menu window 204 andthe panoramic image is fixed, the menu window 204 is moved along withthe panoramic image. On the display panel 130, the user thus watches themenu window 204 moving to the right.

Subfigure (b) in FIG. 14 illustrates a display image 200 e as it isturned 45 degrees to the left from the initial display state. In orderto display the display image 200 e, the user may turn his or her head 45degrees to the left. Alternatively, the 45-degree left turn may beaccomplished using the left analog stick 77 b being tilted twice to theleft or by the user turning the head 22.5 degrees to the left followedby the left analog stick 77 b being tilted once to the left. Comparedwith Subfigure (a) in FIG. 14, the menu window 204 appears to be movedfurther to the right.

Subfigure (a) in FIG. 15 illustrates a display image 200 f as it isturned to the left from the initial display state until the positionchange reference angle is just about to be reached. The position changereference angle is set here to be such that the information elements donot entirely disappear from the screen.

This requires that the position change reference angle be set to besmaller than the viewing angle of the HMD 100. If the position changereference angle is equal to or larger than the viewing angle, theinformation elements could entirely disappear from the screen. Theposition change reference angle in the horizontal direction should thusbe set to be smaller than 100 degrees because the viewing angle of theHMD 100 in the horizontal direction is 100 degrees.

Experiments with various settings of the position change reference angleled the inventors to conclude that a position change reference angle setto be extremely small compared with the viewing angle of the HMD 100gives the impression of the display elements more or less tracking thevisual line. After the experiments, the position change reference anglewas set to be approximately half the viewing angle of the HMD 100. Thisangle setting was found to allow both display elements and the panoramicimage in the background to be integrally recognized and to provide auser interface enabling the display elements to be visible. In theembodiment, the position change reference angle (first angle) is set tobe 60 degrees, which is close to half (50 degrees) the viewing angle ofthe HMD 100.

Subfigure (b) in FIG. 15 illustrates a display image 200 g with theinformation elements turned to the left from the initial display statewithin the panoramic image by the second angle (60 degrees), which isthe return angle. When the amount of change in the visual line in thisdirection reaches the first angle (60 degrees) starting from the initialdisplay state, the image generating section 34 rearranges the relativepositional relation between the information elements and the panoramicimage to a position moved by the second angle in the current direction.At this point, the second angle is set to be the same as the firstangle. After the rearrangement, the visual line direction determiningsection 32 monitors changes in the movement of the visual line directionin reference to the visual line direction determined at the point of therearrangement.

Whereas Subfigure (a) in FIG. 15 illustrates only part of the menuwindow 204, Subfigure (b) in FIG. 15 illustrates the entire menu window204 being displayed again in its entirety. This facilitates menuoperation of the menu window 204 by the user. And as illustrated inSubfigure (a) of FIG. 15, display control is performed in such a mannerthat the menu window 204 will not disappear completely from the screen.This allows the user to operate when desired the menu window 204 withoutlosing sight of it while enjoying the panoramic image.

The foregoing paragraphs explained the example in which the first angleas the position change reference angle is the same as the second angleas the return angle. If the first angle and the second angle areidentical and if the visual line direction is changed by the first anglefrom the initial display state, the information elements are rearrangedto the center position at which the information elements were displayedin the initial display state as illustrated in Subfigure (b) of FIG. 15.Thus if the user turns his or her head further to the left in thehorizontal direction, the menu window 204 moves from the center to theright. That means the menu window 204 is displayed somewhat to theright, which is slightly uncomfortable to look at.

Explained below is an example in which the first angle as the positionchange reference angle is set to be larger than the second angle as thereturn angle. The second angle is set to be 60 degrees, and the firstangle is set to be 45 degrees.

Suppose now that Subfigure (b) in FIG. 14 illustrates the display image200 e as it is turned to the left from the initial display state untilthe first angle (45 degrees) is just about to be reached. If the visualline in this direction is further moved so that the amount of its changereaches the first angle (45 degrees), the image generating section 34rearranges the relation positional relation between the informationelements and the panoramic image to a position moved by the second angle(60 degrees) in the current direction. Because the second angle is setto be larger than the first angle, the rearranged information elementsare returned past the center position of the initial display state andthus displayed to the left of the center position. Following therearrangement, the visual line direction determining section 32 monitorschanges in the movement of the visual line direction in reference to thevisual line direction determined at the point of the rearrangement.

FIG. 16 illustrates a display image 200 h with the information elementsturned to the left at the second angle (60 degrees) within the panoramicimage. When the amount of change in the visual line in this directionreaches the first angle (45 degrees) from the initial display state, theimage generating section 34 rearranges the information elements to aposition returned in the current direction by the second angle (60degrees) within the panoramic image.

With the second angle set to be larger than the first angle, if the userkeeps moving his or her visual line direction further to the left, themenu window 204 gradually moves to the right toward the center. Thismakes it possible to enhance the visibility of the information elementswhile allowing the user to recognize the integrity of the displayelements and the panoramic image in the background. If the first angleis set to be larger than half the second angle, the amount of movementof the display elements in the panoramic image is prevented fromincreasing when the display elements are rearranged.

It was explained above that when the change of the visual line in thehorizontal direction reaches the position change reference angle (firstangle), the arrangement of the information elements in the horizontaldirection is changed by the return angle (second angle). The sameapplies to the vertical direction. When the change of the visual line inthe vertical direction reaches a position change reference angle in thatdirection, display control may be performed in such a manner that thearrangement of the information elements in the vertical direction ischanged by a return angle in that direction. The position changereference angle in the vertical direction may be set to be smaller thanthe position change reference angle in the horizontal direction. Forexample, the position change reference angle in the vertical directionmay be set to be 30 degrees.

Examined next is where information elements are to be initiallydisplayed. One technique involves positioning the information elementsnear the center of the currently displayed image whenever the elementsare displayed as illustrated in Subfigure (a) of FIG. 13. This techniqueis effective because it is easy to understand intuitively for the user.

Another technique may involve setting beforehand candidate displaypositions for the information elements in the virtual sphere. In thecoordinate system of the virtual sphere, the horizontal referencedirection on the horizontal plane is assumed to be zero degree. On thatassumption, candidate rotation angles are set to be 60 degrees, 120degrees, 180 degrees, 240 degrees, and 300 degrees in the horizontaldirection; and candidate inclination angles are set to be 0 degree, ±30degrees, and ±60 degrees in the vertical direction. When the informationelements are displayed, the candidate rotation angle and the candidateinclination angle closest to the visual line direction may be extracted,and the information elements may be displayed in a manner centering onthe extracted rotation angle and inclination angle. Where the initialposition candidates are determined in advance, the image generatingsection 34 need only select the appropriate initial position from amongthe candidate positions. This technique simplifies display control.

While the present invention has been described in conjunction with aspecific embodiment given as an example, it should be understood bythose skilled in the art that the above-described composing elements andvarious processes may be combined in diverse ways and that suchcombinations, variations and modifications also fall within the scope ofthis invention.

It was explained above with reference to Subfigure (a) of FIG. 14 thatthe display image 200 d and menu window 204 are displayed turned 22.5degrees to the left from the initial display state and that the22.5-degree left turn may be achieved either by the user turning his orher head or by the left analog stick 77 b being operated. In avariation, the left analog stick 77 b may be operated to cancel therelative positional relation between the information elements and thepanoramic image. In this case, the information elements may be displayedat the center as illustrated in Subfigure (b) of FIG. 13 in the imagefollowing the witch by operation of the left analog stick 77 b. This ismade possible because image continuity is broken by a jump of thedisplay image when it is switched by operation of the left analog stick77 b. In such a case, display control of the information elements isavailable following the unlocking of the relative positional relationbetween the information elements and the panoramic image.

It was also explained in connection with the embodiment that displaycontrol of the information elements is performed using the positionchange reference angle and the return angle. In a variation, displaycontrol of the information elements may be carried out in accordancewith the information element type. For example, if the system issues animportant warning to the user, the image generating section 34 mayalways arrange the warning at the center of the display image as theinformation element.

The image generating section 34 may implement a mirror function as a VRapplication function. The mirror function in this context is thefunction of including an image opposite to the visual line directioninto the display image. What is included into the display image is notan image that is the exact opposite to the visual line direction but animage obtained by reversing only the rotation angle of the visual linedirection, i.e., by turning the rotation angle of the visual linedirection 180 degrees while leaving the inclination angle intact. Thismirror function allows the user to view the image behind his or her backat the same height.

REFERENCE SIGNS LIST

-   1 Information processing system-   6 Input device-   8 Virtual camera-   10 Information processing apparatus-   12 Processing device-   14 Output control device-   20 Sensor information acquiring section-   22 Captured image acquiring section-   24 Instruction acquiring section-   30 Motion detecting section-   32 Visual line direction determining section-   34 Image generating section-   36 Image providing section-   40 Image storing section-   42 Information element storing section-   100 Head-mounted display device (HMD)-   102 Output mechanism section-   104 Wearing mechanism section-   106 Wearing band-   108 Enclosure-   110 Light-emitting marker-   120 Control section-   122 Storage section-   124 Attitude sensor-   126 Microphone-   128 Communication control section-   130 Display panel-   132 Audio output section

INDUSTRIAL APPLICABILITY

The present invention may be implemented in technical fields where thehead-mounted display device is used.

The invention claimed is:
 1. An information processing apparatuscomprising: a detecting section configured to detect the attitude of ahead-mounted display device worn on the head of a user; an imagegenerating section configured to generate an image in accordance withthe attitude of the head-mounted display device detected by thedetecting section; and an image providing section configured to providethe head-mounted display device with the generated image, wherein theimage generating section includes an information element superimposed onthe generated image, and when the attitude of the head-mounted displaydevice is changed by less than a first angle during display of the imageincluding the information element on the head-mounted display device,the image generating section displays the information element in such amanner that the information element moves along with the image, and ifthe attitude of the head-mounted display device reaches the first angle,the information element is moved to a position by a second angle in anopposite direction to a rotation direction of the head-mounted displaydevice.
 2. The information processing apparatus according to claim 1,further comprising: a visual line direction determining sectionconfigured to determine a visual line direction in accordance with theattitude of the head-mounted display device detected by the detectingsection, wherein the image generating section generates an image basedon the determined visual line direction, and the image generatingsection displays the information element in such a manner that theinformation element moves along with the image if the movement of avisual line in a predetermined direction is smaller than the firstangle, the image generating section further displaying the informationelement in a position moved by the second angle in the predetermineddirection in the image if the movement of the visual line in thepredetermined direction reaches the first angle.
 3. The informationprocessing apparatus according to claim 2, wherein the image generatingsection performs display control of the information element inaccordance with a rotation angle of a visual line in a horizontaldirection.
 4. The information processing apparatus according to claim 1,wherein the first angle is smaller than a viewing angle of thehead-mounted display device.
 5. The information processing apparatusaccording to claim 1, wherein the second angle is larger than the firstangle.
 6. An image generating method comprising: detecting the attitudeof a head-mounted display device worn on the head of a user; andgenerating an image in accordance with the detected attitude of thehead-mounted display device, wherein the image generating includes aninformation element superimposed on the generated image, and when theattitude of the head-mounted display device is changed by less than afirst angle during display of the image including the informationelement on the head-mounted display device, the image generatingdisplays the information element in such a manner that the informationelement moves along with the image, and if the attitude of thehead-mounted display device reaches the first angle, the informationelement is moved to a position by a second angle in an oppositedirection to a rotation direction of the head-mounted display device. 7.A non-transitory computer readable medium having stored thereon aprogram for a computer, the program comprising: detecting, by adetecting section, the attitude of a head-mounted display device worn onthe head of a user; and generating, by an image generating section, animage in accordance with the detected attitude of the head-mounteddisplay device, wherein the image generating includes an informationelement superimposed on the generated image, and when the attitude ofthe head-mounted display device is changed by less than a first angleduring display of the image including the information element on thehead-mounted display device, the image generating section displays theinformation element in such a manner that the information element movesalong with the image, and if the attitude of the head-mounted displaydevice reaches the first angle, the information element is moved to aposition by a second angle in an opposite direction to a rotationdirection of the head-mounted display device.